Comparison of Methods for Constrained CAT Item Selection in Classification Accuracy and Consistency

The method developed for handling multiple constraints, namely the maximum priority index (MPI) method (Cheng & Chang, under review), was shown to be very effective in a simulation study with a national computerized placement test. To be more specific, compared with the currently widely used weighted deviation modelling (WDM) method, the MPI method reduces constraint violations substantially while being comparable in terms of measurement precision. The current study compares the MPI method with the WDM method in terms of classification accuracy and consistency. The maximum information and randomized item selection methods are also included to provide baseline measures. Constraint management statistics and overall measurement precision indices are calculated. The results show that the MPI method is a very good candidate for item selection of CAT programs leading to better constraint management and smaller test overlap rate while maintaining the same level of classification accuracy and consistency as the WDM method does.